Repository navigation
[XNNPACK] Keep cpuinfo initialized for the lifetime of the EP - #34228
Open
Nopthakorn Kutawan (im-noon) wants to merge 1 commit into
Open
Nopthakorn Kutawan (im-noon) wants to merge 1 commit into
Nopthakorn Kutawan (im-noon) wants to merge 1 commit into
Conversation
XNNPACK calls cpuinfo_initialize()/cpuinfo_deinitialize() around its hardware config lookups in every operator's reshape/setup, and cpuinfo is reference counted. On Apple and Android, PosixEnv doesn't hold a cpuinfo reference and the CPUIDInfo singleton is created lazily (by the NHWC transformer, the thread pool or some CPU kernels). In a single-threaded session where nothing has created it yet, e.g. an ORT format model in a minimal build whose nodes are all taken by XNNPACK, every XNNPACK op re-detects the CPU: dozens of sysctl calls on macOS/iOS, about 35 us per op on an Apple M-series CPU. A chain of 20 3x3 Conv nodes (32 channels, 14x14), Fixed-style ORT format, minimal build, 1 thread: 1.46 ms before, 0.76 ms after (CPU EP: 0.95 ms). Create the CPUIDInfo singleton in the XNNPACK EP constructor so cpuinfo stays initialized. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
A provider-level regression test is missing: existing tests do not detect removal of the cpuinfo retention fix because they either pass without it or never construct the provider.
1 open issue
What changed in this PR
This PR avoids repeated CPU detection in XNNPACK by eagerly creating ONNX Runtime’s CPUIDInfo singleton. It targets inference overhead in single-threaded, XNNPACK-only sessions where the singleton might otherwise remain uninitialized.
Changes:
- Initialize
CPUIDInfoin the XNNPACK execution provider constructor to retain cpuinfo’s reference. - Document why lazy initialization can cause repeated hardware detection, especially in minimal builds on Apple platforms.
| File | Description |
|---|---|
onnxruntime/core/providers/xnnpack/xnnpack_execution_provider.cc |
Includes the CPUIDInfo header, initializes the singleton, and explains the lifetime requirement. |
Review effort: Balanced
| // process lifetime, but it is created lazily (by graph optimizers, the thread pool or some CPU kernels), so a | ||
| // single-threaded session whose nodes are all taken by XNNPACK, e.g. an ORT format model in a minimal build, may | ||
| // never create it. | ||
| CPUIDInfo::GetCPUIDInfo(); |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Create the
CPUIDInfosingleton in the XNNPACK EP constructor so cpuinfo stays initialized for the lifetime of theprocess.
Motivation and Context
XNNPACK calls
cpuinfo_initialize()/cpuinfo_deinitialize()around its hardware config lookups, which run in everyoperator's reshape/setup, and cpuinfo is reference counted. ORT normally keeps a reference through the
CPUIDInfosingleton, but on Apple and Android
PosixEnvdoesn't hold one andCPUIDInfois created lazily (by graph optimizers,the thread pool or some CPU kernels). In a single-threaded session where none of those has run, e.g. an ORT format
model in a minimal build whose nodes are all taken by XNNPACK, nothing holds a reference, so every XNNPACK operator
re-detects the CPU from scratch. On macOS/iOS that is dozens of
sysctlcalls per operator; sampling shows__sysctlas the hottest function, and it costs about 35 us per XNNPACK op on an Apple M-series CPU.Measured with
onnxruntime_perf_test, extended minimal build (upstream main), macOS arm64, 1 thread, P50,Fixed-style ORT format model made of a chain of 20 3x3
Convnodes (32 channels):Before the fix XNNPACK is slower than the CPU EP on the small model; after it, faster. Models where CPU kernels also
run (and create
CPUIDInfo), and full builds that run the graph optimizers, are unaffected: torchvisionMobileNetV2/V3 and ResNet18 timings are unchanged within noise in both builds.
Testing: the XNNPACK unit tests in
onnxruntime_provider_test(20 tests) and the Conv/Pool/MatMul/Gemm/Softmax/Resizeop tests (1798 tests) pass in a full build with XNNPACK;
onnx_test_runnerresults on the ORT format test models areunchanged in the minimal build.
🤖 Generated with Claude Code